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q ' Aims. Lya emitters (LAEs) can be detected out to very high redshifts during the epoch of reionization. The evolution of the LAE 

J_h . luminosity function with redshift is a direct probe of the Lya transmission of the intergalactic medium (IGM), and therefore of the 

. IGM neutral-hydrogen fraction. Measuring the Lya luminosity function (LF) of Lya emitters at redshift z = 7.7 therefore allows us 

■ to constrain the ionizing state of the Universe at this redshift. 

Methods. We observed three 7!5x7!5 fields with the HAWK-I instrument at the VLT with a narrow band filter centred at 1.06 yum and 
targeting Lya emitters at redshift z ~ 7.7. The fields were chosen for the availability of multiwavelength data. One field is a galaxy 
cluster, the Bullet Cluster, which allowed us to use gravitational amplification to probe luminosities that are fainter than in the field. 
The two other fields are subareas of the GOODS Chandra Deep Field South and CFHTLS-D4 deep field. We selected z = 7.7 LAE 
' candidates from a variety of colour criteria, in particular from the absence of detection in the optical bands. 

Results. We do not find any LAE candidates at z = 7.7 in ~ 2.4 x 10 4 Mpc 3 down to a narrow band AB magnitude of ~ 26, which 
, allows us to infer robust constraints on the Lya LAE luminosity function at this redshift. 

Conclusions. The predicted mean number of objects at z = 6.5, derived from somewhat different luminosity functions of Hu et al. 
(2010), Ouchi et al. (2010), and Kashikawa et al. (201 1) are 2.5, 13.7, and 1 1.6, respectively. Depending on which of these luminosity 
functions we refer to, we exclude a scenario with no evolution from z = 6.5 to z = 7.7 at 85% confidence without requiring a strong 
change in the IGM Lya transmission, or at 99% confidence with a significant quenching of the IGM Lya transmission, possibly from 
a strong increase in the high neutral-hydrogen fraction between these two redshifts. 

Key words. Methods: observational - Techniques: image processing - Galaxies: high-redshift - Galaxies: luminosity function, mass 
function - early Universe - dark ages, reionization, first stars 
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6.56 dHu et al.ll2002l) . spectacular progress has been made in as- 
sembling large samples of high-redshift objects. The two main 
techniques for finding high-redshift galaxies is to look either for 
strong absorption breaks in the Ly a forest in broad band photom- 
etry (Lyman break galaxies - LBGs) or for a photometric excess 
in narrow band (NB) filters due to the Lya line (Lye emitters - 
LAEs). In the latter case, the NB filters are usually selected to co- 
incide with regions of low OH emission of the night sky, leading 
to discrete redshift values. SuprimeCam on the Subaru Telescope 
has revolutionized the field by enabling larg e samples of LAEs 
to be furnishe d at z = 5.7 and z = 6.5 (lOuchi et al.l l2Q10t 
iHu et alj|2010l and references therei n). The largest samp les of 
LBGs have been recently assembled dBouwens et al.l20"Tlh from 
HST observations after the successful installation of the Wide 
Field Camera3 (WFC3) in May 2009, but LBGs can also be 
found from the ground with 8-10 m te lescopes equipped with 
efficient near infrared (NIR) cameras (ICaste llano et al.l 120101) . 
Quasars at high-redshift are also found using the Lyman break 
technique in multi-colour datasets over very wide fields. Most 
of the quasars at z > 6 have been discovered in the Sloan 
Digital Sky Survey ( Fan et al. 20061) a nd from a targeted pro- 
gramme at CFHT (IWillott et all 120101) . Finally, a few gamma 
ray burs ts (GRBs) have be en discovered at very high redshift 
(see e.g. lTanvir et al.ll2009l for an example of a GRB at redshift 
8.2), nicely complementing the other methods by probing the 
faint end of the luminosity function. 

Combined with observations of the cosmic microwave back- 
ground (CMB), the recent discovery of large samples of objects 
at high redshift allows astronomers to build a comprehensive pic- 
ture of the Universe during the reionization epoch when it was 
500 Myr to 1 Gyr old. Polarizati on measurements of the CMB 
from WMAP dLarson et al.ll201ll) show a large optical depth due 
to Thompson scattering of electrons in the early Universe, sug- 
gesting that the reionization started at z ~ 10.5 + 1 .2. Conversely, 
the strong increase o f the optical depth in the Lya fo rest of 
high-redshift quasars dBecker et al . 11200 lllFan et al.ll2006l) above 
~ 8500 A is a likely indicator that reionization was mostly com- 
plete at a redshift of about 6. How and at what pace the reioniza- 
tion process has taken place in the [6-10] redshift range is more 
difficult to establish from observations, and is still a matter of 
debate. A compilation of the most recent results and constraints 
on the neutral-hydrogen fraction of the Universe between red- 
shifts 5 and 11 fr om various probes is shown in figure 23 of 
lOuchi etal.ld2010l) . 

It has been proposed for a long time to use the Lya trans- 
mission by the intergalactic medium (IGM) as a prob e of its 
ioniza tion state during the reionization epoch (see e.g. ISantosi 
120041) . hence the strong emphasis recently put on Lya emission 
of LBGs and LAEs as more and more of these objects become 
available. Follow-up observations of high-z LBGs at z > 6 is 
now underway to detect the Lya line in emission in spectroscopy. 
I Stark et all (1201 ll) measure an increasing fraction of LBGs with 
strong Lya emission from z ~ 3 to z ~ 6, and conjecture that 
Lya emission should remain strong at higher redshifts unless 
the neutral- hydrogen fract i on of the IGM suddently increases. 
Conversely, iFontana et al.l d2010h report a low fraction of Lya 
emitters in a sample of z > 6.5 LBGs. These are preliminary 
results based on still modest spectroscopic samples, and it is ex- 
pected that ongoing and new observations will clarify the situa- 
tion in a near future. 

Another observational method of probing the Lya IGM 
transmission is to study the evolu tion of the LAE lumi- 
nosity function (LF) with r edshift. lOuchi etaO (1201 Oh and 
iKashikawa etaD (I2006L I20ITI) infer from their observations that 



the evolution of the Lya LAE LF between z — 5.7 and z — 6.5 
can be attributed to a reduction of the IGM Lya transmission of 
the order of 20%, which can in turn be attributed to a neutral- 
hyd rogen fraction xht of the order of 20% at z = 6.5 (see 
e.g. lOuchi et"ai1 l2010t> . Various models are elaborated to re- 
produce this claim, which has generated considerable interest 
(see e . g.lKobavashi et ai1|2010l:lDaval et af]l201 ll : lLaursen et al.l 
1201 lb iDiikstra & Wvithq 120101). However, th e un iversality of 
the z - 6.5 LF from IKashikawa et alj d2006h and lOuchi et al.l 
d2010h has recently been questioned. Hu et al. (1201 C ) report sig- 
nificantly different LF parameters from the observations and 
analysis of a spectrosc opically confirmed sam ple of NB se- 
lected LAEs. Similarly, iNakamura et al.l (1201 ll) report signif- 
icantly lower number counts that they tentatively attribute to 
cosmic variance. Differences in the selection criteria and in ex- 
trapolations of the spectroscopic samples to photometric sam- 
ples might partly explain the discrepancies b etween the vari- 
ous Ly a LAE LFs available in the literature: IKashikawa et al.l 
d201ll) have carried out extended spectroscopic confirmation of 
their earlier photometric sample, resultin g in luminosity func- 
tions closer to the ones of IHu et al.l d2010l) . ICa'ssata et al.l ( 1201 lh 
report the results from a pure spectroscopic sample of (mostly) 
serendipitous Lya emitters found in deep spectroscopic samples 
with VIMOS at the VLT ; this sample is consistent with a con- 
stant LAE luminosity function from z ~ 2 to z ~ 6.6 as report ed 
in the literature before the recent results from IHu et al.l (120 101) . 

The current situation at z ^ 6 is therefore unclear, with 
somewhat contradictory observational results. This hampers the 
validation of the reionization models and of our understanding 
of this key epoch of the Universe. The discrepancy limits how 
well we can understand reionisation during this key epoch of the 
Universe. New data at z ~ 6 will help in resolving the current 
contention between observational results, while data at higher 
redshifts can bring new constraints at still poorly explored red- 
shifts. In view of the strong interest in studying Lya emission at 
high redshift s, searching LAEs at z > 7 is underway from var- 
ious g roups dHibon etal.ll2010t iTilvi etal.ll20Tot iNilsson et all 
|2007|) . Finding z ~ 7 objects is not only interesting for probing 
the reionization epoch, but also for assessing the physical prop- 
erties of these objects, which in turn allow constraining how and 
when they formed. Due to the extreme faintness of these very 
high-redshift objects, deriving t heir properties can onl y be done 
statistically over large samples dBouwens et al 1 120111) " or on in- 
dividual objects that are gravitationally amplified. For instance, 
iRichard et al.l d201 ll) infer a redshift of formation of 1 8 + 4 for a 
gravitationally amplified object at z = 6.027. 

This paper presents new results on the Lya LAE LF at z = 
7.7, from observations carried out at the VLT with the HAWK-I 
instrument. This paper is organized as follows. In section |2] we 
describe the observations and the data reduction in section [3] In 
section|4]we describe our selection procedure of the z = 7.7 LAE 
candidates. In section|5]we present the constraints that we infer 
from our results on the z — 7.7 Lya LAE LF, before discussing 
our results in section [6] 

We use AB magnitudes throughout this paper. We assume a 
fiat ACDM model with £l M = 0.30 and H = 70 km s _1 Mpc" 1 . 



2. Observations 

This work is primarily based on extremely deep NIR imaging 
data obtained with HAWK-I at the VLT, using an NB filter at 
1.06 /im (hereafter referred to as NB1060). Thanks to its wide 
field of view (7!5x7'5), excellent throughput and image quality, 
HAWK-I is ideally suited to searching for faint NIR objects such 
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as very high-redshift galaxies. The main data set was obtained 
through a dedicated ESO large programme between September 
2008 and April 2010. In addition, we include in our analysis 
HAWK-I science verification NB data taken in 2007. We also 
make use of various optical and NIR broad band data, publicly 
available and/or from our own large programme. 

2.1. Fields 

In preparing the proposal, we carefully balanced the relative 
merits of blank fields and cluster fields. While gravitational am- 
plification of background sources by foreground massive galaxy 
clusters allows us to probe luminosities that are intrinsically 
fainter than in the field, this is at the expense of areal coverage 
due to space distortion. The relative merits of blank and clus- 
ter fields depend on the shape of the luminosity function (LF) 
of the objects that are being searched, and on the properties of 
the obs ervations such as field of view, integatio n time and over- 
heads dMaizv et alJ l2010t iRichard et aT1l2008l) . From the Lya 
LAE LF at z = 6.5 that was available at the time of proposal 
preparation, we computed that either type of fields should yield 
approximately the same number of targets, while probing dif- 
ferent (unlensed) luminosity ranges. We also analysed the bal- 
ance between wide-shallow and narrow-deep survey strategies. 
For a total time of about 100 hrs (in the NB filter only), it was 
deemed that observing four fields in total would be optimal in 
terms of high-z LAE yield, while mitigating the effects of cos- 
mic variance. Operational constraints, such as the distribution 
of the fields in right ascension, were additionally taken into ac- 
count when selecting the fields. Our selected fields were Abell 
1689 (13*1 l m 30', -01°20'35", J2000) and 1E0657-56 (Bullet 
Cluster) (06 / '58"'29 i , -55°57'16", J2000) for the cluster fields, 
the northern half of the GOODS-S field (03 /, 32" , 29 s , -27°44'42", 
J2000) and a subarea of the one square degree CFHTLS-D4 field 
(22*16 m 38 \ -17°35'41", J2000) for the two blank fields. 

For Abell 1689, although an extensively studied field, it 
proved hard to assemble a consistent multiwavelength dataset 
covering the full 7:5x7:5 HAWK-I field of view. This field is 
therefore not included in the present analysis and it will be anal- 
ysed separetely. The Bullet Cluster is a massive merging cluster 
that allowed the first direct empirical proof of the existence of 
dark matter by the combination of strong and weak-lensing anal- 
yses (IClowe et al]l2006tlBradac et al.ll2006l) . Both clusters have 
well-constrained mass models and provide a lens magnification 
of at least a factor of 1 .2 over 50% of the HAWK-I field of view 
(see Figure[U. The GOODS-S and CFHTLS-D4 field were cho- 
sen for the wealth of multiwavelength data, in particular deep 
optical data, publicly available. For the CFHTLS-D4 field, we 
chose the location of the HAWK-I observations where NIR data 
were available 1 (Bielby et al., in preparation), and paying atten- 
tion to avoiding the brightest stars present in this field. Figure Q] 
shows the finding charts corresponding to our observations in- 
side the CFHTLS-D4, GOODS-S and Bullet Cluster fields. 

TableQ]summarizes the various observations made as part of 
our large programme on each of the three fields considered in the 
present analysis. Figure|2]shows the overall transmission curves 
of the HAWK-I broad band and NB filters corresponding to these 
observations. Table[2]summarizes the main ancillary broad band 
data used in this work. Our large programme data consists of 
more than 110 hrs of on-sky integration time, of which ~80 hrs 
are NB1060 data. 



1 From the CFHT WIRCam Deep Survey (WIRDS), see 
http : //terapix . iap . fr/rubrique . php?id_rubrique=261 
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Fig. 2. Transmission curves of the HAWK-I broad band and 
narrow band filters corresponding to the observations made as 
part of our large programme. The inset shows the profile of the 
NB1060 filter. 



2.2. The HAWK-I NB 1060 data 

HAWK-I is a 7:5x7:5 NIR (0.97-2.31 jum ) imager installed on 
the ESO VLT UT4. It is equipped with four 2048 x 2048 pixels 
Hawaii-2RG detectors, separated by 15" wide gaps. The pixel 
scale is 071065. The NB1060 filter has a central wavelength 
of 1062 nm, a full width at half maximum (FWHM) of AA ~ 
100 A, and is designed to match a region of low OH emission 
from the night sky. The filter width samples Lya emission in the 
redshift range z = [7.70 - 7.78]. A detector integration time of 
300 s is used for all the NB1060 images, ensuring background 
limited performance. Random telescope offsets within a box of 
20" for the blank fields and 25" for the cluster fields is used for 
dithering. For each field, the NB1060 data are acquired over two 
epochs separated by one year, allowing us to discard transient 
objects that could be detected in a one-epoch stack, and not in 
the other. 

The instrument had a thermal leak at the beginning of the first 
semester, which approximately doubled the total background in 
the NB1060 filter. After a technical intervention on the instru- 
ment after a few months of operations, the background returned 
to its nominal value, and only the observations of the CFHTLS- 
D4 field were affected. We were granted compensatory time that 
allowed us to recover the expected limiting magnitude but at the 
expense of unbalanced limiting magnitudes (by ~ 0.5 mag) for 
the first and second epoch observations. For the two other fields 
the limiting magnitudes between the two epoch observations are 
within ~0.15 magnitude. 

In total, after image selection discarding images with poor 
image quality or too high background, the final stacks used in 
this analysis total integration times in the NB1060 filter of 26.7 
hrs for the CFHTLS-D4 field, 3 1 .9 hrs for GOODS-S (including 
science verification data) and 24.8 hrs for the Bullet Cluster (see 
Tabled). 
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Fig. 1. Images of the CFHTLS-D4 (left), GOODS-South (centre), and Bullet Cluster (right) fields as in the final NB1060 image 
stacks. The inner and outer black contours on the Bullet Cluster image represent the regions where the gravitational amplification is 
respectively > 2.5 (Am < -1) and > 1.2 (Am < -0.2) for a source at redshift z = 7.7. 

Table 1. HAWK-I narrow band and broad band observations from our large programme 



Field 


Filter 


Exposure time 


Seeing 


Limiting magnitude" 






(hrs) 


(") 




CFHTLS-D4 


NB1060 


26.7 


0.53 


26.65 


CFHTLS-D4 


J 


5.0 


0.46 


26.55 


CFHTLS-D4 




0.83 


0.50 


24.6 


GOODS-S 


NB1060 


31.9 


0.58 


26.65 


GOODS-S 


J 


3.3 


0.44 


26.55 


BULLET CLUSTER 


Y 


6.1 


0.59 


26.50 


BULLET CLUSTER 


NB1060 


24.8 


0.55 


26.50 


BULLET CLUSTER 


J 


6.5 


0.49 


26.55 


BULLET CLUSTER 


K s 


3.75 


0.45 


25.45 



Notes. (a) 3cr aperture corrected limiting magnitude. 



2.3. Other imaging data 

In addition to the NB1060 data, we performed dedicated broad 
band HAWK-I observations within our large programme to com- 
plement, on a case-by-case basis, the broad band data that were 
available elsewhere (see Table[TJ. 

For the CFHTLS-D4 field, we had access to the very deep 
CFHTLS optical data and to the moderately deep NIR WIRDS 
data, but not as deep as our NB1060 data. We therefore took ad- 
ditional J and K s data to improve the detection limit in these 
bands. For the Bullet Cluster, in the absence of well-established 
datasets in the optical and NIR bands, particularly over the full 
HAWK-I field of view, we devoted a significant fraction of the 
time on this field to get additional data in the Y, J, and K s bands. 
We note that the HAWK-I Y filter bandpass includes, at its very 
red edge, the NB1060 filter bandpass. For this field we there- 
fore secured a coherent and self-consistent dataset. We also used 
IMAC S BVR imag es of the field obtained at the Magellan tele- 
scope dClowe et alil2006l) . For the GOODS-S field, we devoted a 
few hours of observations in the / band filter to reach a limiting 
magnitude fainter than that achieved with the public ISAAC im- 
ages. We also used a very deep F-band image of this field from 
a separate HAWK-I large programme (181.A-0717) led by one 
of us. 



3. Data reduction 

All reduced image stacks and ancillary data products (e.g. 
weight maps, etc.) from our large programme are currently avail- 



able upon request and will be made public through the ESO 
archive, as part of the Phase 3 process 2 . We detail in this section 
the data reduction procedures that have been used to generate 
these high-level data products. 

3.1. Overview 

We use a mix of IRAF 3 and AstrOmatic 4 routines for the data 
reduction, allowing us to control the reduction process step by 
step. All single quadrant frames from the HAWK-I mosaic ar- 
ray are reduced similarly and independently until the very final 
steps. After a first pass at the sky subtraction, all images are char- 
acterized in terms of PSF and photometric quality, allowing us 
to identify and remove low-quality images that would degrade 
the final products. Depending on the field, between 3% and 13% 
of the NB1060 images are discarded, mostly when the image 
quality is worse than 1". This is the case for a fair fraction of 
the GOODS-S science verification images, for some of our large 
programme images that have been executed but not validated 
by the service observers, and, more rarely, for some fully val- 
idated images. A multi-pass sky subtraction is then performed, 
improving the quality of the masking of the objects at each pass. 



2 http://www.eso.org/sci/observing/phase3.html 

3 IRAF is distributed by the National Optical Astronomy 
Observatory, which is operated by the Association of Universities 
for Research in Astronomy, Inc., under cooperative agreement with the 
National Science Foundation. 

4 See http://www.astromatic.net/ 
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6.5 to z = 7.7 : evidence for the epoch of reionization ? 



Field Filter Instrument Seeing Limiting magnitude Reference 



(") 


CFHTLS-D4 


u* 


CFHT/Megacam 


0.92 


27.40° 


1 


CFHTLS-D4 


g' 


CFHT/Megacam 


0.85 


28.20° 


1 


CFHTLS-D4 


r' 


CFHT/Megacam 


0.77 


28.00° 


1 


CFHTLS-D4 


V 


CFHT/Megacam 


0.73 


27.45° 


1 


CFHTLS-D4 


z 


CFHT/Megacam 


0.72 


26.60° 


1 


CFHTLS-D4 


J 


CFHT/WIRCam 


0.70 


25.10° 


2 


CFHTLS-D4 


H 


CFHT/WIRCam 


0.67 


24.70° 


2 


CFHTLS-D4 


K s 


CFHT/WIRCam 


0.65 


24.70° 


2 


GOODS-S 


B (F435W) 


HST/ACS 


0.12 


27.95° 


3 


GOODS-S 


V(F606W) 


HST/ACS 


0.11 


28.10° 


3 


GOODS-S 


I(F775W) 


HST/ACS 


0.10 


27.55° 


3 


GOODS-S 


Z (F850LP) 


HST/ACS 


0.10 


27.25° 


3 


GOODS-S 


J 


VLT/ISAAC 


0.50 


26.00° 


4 


GOODS-S 


H 


VLT/ISAAC 


0.53 


25.35° 


4 


GOODS-S 


a; 


VLT/ISAAC 


0.47 


24.65° 


4 


GOODS-S 


Y 


VLT/HAWK-I 


0.55 


27.20° 


5 


GOODS-S 


NB1060 


VLT/HAWK-I 


0.70 


25.65° 


6 


GOODS-S 


K s 


VLT/HAWK-I 


0.40 


26.00° 


5 


BULLET CLUSTER 


R-Bessel 


Magellan/IMACS 


0.60 


28.00" 


7 


BULLET CLUSTER 


B (F435W) 


HST/ACS 


0.10 


27.95'' 


8 


BULLET CLUSTER 


V(F606W) 


HST/ACS 


0.10 


28.15* 


8 


BULLET CLUSTER 


I (F775W) 


HST/ACS 


0.10 


28.55* 


8 


BULLET CLUSTER 


I (F814W) 


HST/ACS 


0.10 


28.30* 


7 


BULLET CLUSTER 


Z (F850LP) 


HST/ACS 


0.10 


27.90* 


8 



Notes. (a) 3<r aperture corrected limiting magnitude. ( *' 3cr aperture (© = 0760) limiting magnitude. 

Re ferences. (1) CFHTLS T0006 release; (2) WIRDS T000 2 release; (3) GOOD S V ersion 2.0 HST ACS Im aging Data; (4) lRetzlaff etHI j20Tch : 
(5) ICastellanoetalJ<201Cl) : (6) ESO Prog-Id 60.A-9284; m lClowe et at] < l2006h : (8) lGonzalez et al.U2009h . 



Images are then scaled to account for photometric variations and 
registered to a common reference frame. The final stacks are pro- 
duced by averaging all images with a rejection algorithm using 
kappa-sigma clipping. The NB1060 photometric calibration is 
performed on unsaturated bright stars by interpolating their op- 
tical and 2MASS phot o metric data, following the approach de- 
scribed in iHibon et al.l (120101) . Finally, the final four stacks for 
each quadrant are aligned onto a common astrometric reference. 



3.2. Data processing 

1. Pre-processing. Dark frames and twilight sky flats are com- 
bined into master calibration frames on a nightly basis. There 
are typically eight dark frames and about 30 twilight sky 
flatfield frames per night. No attempt is made to correct for 
the detector's non-linearity, which, according to the HAWK- 
I users manual, is below the 1% level at 75% of the detector 
saturation level. This could affect the accuracy of the photo- 
metric calibration performed on bright stars, and this is ac- 
counted for in the following. 

2. Background subtraction. The most delicate step in the re- 
duction of NIR data is the sky subtraction. With dithered 
images, the classical way of estimating the sky at any par- 
ticular pixel is by building a running sky frame for each 
science frame. This running sky frame is usually computed 
as the median of N s ky frames around the central science 
frame to which it is subtracted. Some care is required, how- 
ever, for this step to be optimally performed. First, the sky 
background varies, even in the NB1060 filter where the sky 
consists of a mixture of faint OH lines, sky continuum, 
and possibly faint thermal background leaking through the 



wings of the filter at wavelengths close to the detector cut- 
off. In addition, the background patterns have structures at 
low spatial frequencies that are changing with time, with 
the strongest changes occurring when the telescope crosses 
meridian. This is attributed to the rotation of the telescope 
pupil with time, with maximum velocity when passing the 
meridian. Therefore, the images used to generate a sky frame 
are carefully selected so as to have similar sky background 
patterns and to be close in time (within fewer than 15 days). 
The images thus selected are further zeroed to their median 
levels and normalized to their pixel to pixel standard devia- 
tion (with rejection of outliers). For each pixel, the median 
of its values in each running list of N s ky frames is computed, 
with kappa-sigma clipping for rejecting outliers. In this step, 
objects are masked (meaning that the values of the pixels 
where objects are detected are not included in the median 
determination) to avoid biasing the estimation of the sky to- 
ward high values. The masking process is initiated on indi- 
vidual sky-subtracted frames (where only the brightest ob- 
jects are detected) and then repeated several times on com- 
bined stacks as described in step # 4. 

3. Bad pixels removal. Once the initial sky subtraction is per- 
formed on each science frame, bad pixel maps are gener- 
ated on a nightly basis. Here again, we use the fact that with 
dithered data an object moves across the detector while bad 
pixels do not. Individual pixel values exceeding +4<x of the 
local standard deviation over more than 70% of the frames 
in a given night are flagged as bad pixels and replaced by 
a linear interpolation of the surrounding pixel values along 
image lines. 

4. Object masking for sky subtraction. After the initial sky 
subtraction step and bad-pixel removal, the images are reg- 
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istered using a first-order astro metric solution and median - 
stacked with rejection of outliers. A mask is then gener- 
ated from all objects detected in this image, together with 
detector regions of poor cosmetics. This mask is then used 
to reprocess the sky frames as described in step #2, af- 
ter which a new image stack is produced. Steps #2 to #4 
are typically repeated three to five times until the back- 
ground around the objects in the final stack is flat. This it- 
erative procedure improves the quality of the sky subtrac- 
tion, which otherwise results in overestimated sky levels no- 
ticeable as dark regions around the bright objects and in 
larger photometric errors. As in step #2, the final sky frames 
are subtracted to the central science frames after zeroing to 
their median values and scaling to their standard deviations. 
Faint low-frequency sky subtraction residuals may still re- 
main at this stage, which are removed with a bi-cubic-spline 
interpolation of a meshed background frame generated by 

SExtractoiEl dBerfin & Arnoutsll 19961) . 

5. Correction of photometric variations. Frame-to-frame 
scaling factors are derived from the number counts mea- 
sured on bright and unsaturated stars detected in each in- 
dividual sky-subtracted frame. These scaling factors account 
for variations of the atmosphere transparency and/or of the 
airmass. Between two to ten stars per quadrant frame are 
typically used and the fluxes derived from the SExtractor 
MAG_AUTO measurements. For all three fields, the varia- 
tions of these frame-to-frame scaling factors are below 10% 
peak-to-peak, with a star-to-star variation within each frame 
of about 1.5%. 

6. Image registration. A relative astrometric solution is com- 
puted for each sky-subtracted frame using Scampi and a 
fourth-order polynomial fit of bright star positions across the 
detector plane. All the resulting images are then resampled 
to a common reference frame with S warp dBertin et al.l2002h 
using a LANCZOS4 interpolation kernel and a pixel scale 
of 071065. The interpolation introduces correlated noise be- 
tween pixels, and this is accounted for when computing the 
signal-to-noise ratio of the object as discussed in section 
13.3.21 The accuracy of the image registration is well within 
one pixel for the whole data set. 

7. Final stacks and weight maps. The final stacks for each 
quadrant are finally produced by averaging with Act rejection 
the individual science frames processed as described. In this 
process, a map identifying the rejected pixels and a sigma 
map are produced. In the latter, sigma (cr) is the standard 
deviation of the N input pixel values, excluding the rejected 
ones, entering into the stacks. The weight maps are then de- 
rived by computing N/cr 2 . 

8. Absolute photometric calibration. The broad band / and 
K s data taken as part of our main programme are photo- 
metrically matched to existing photometric catalogues and 
images of the fields in these filters. We carefully select 
stars in our HAWK-I images for the photometric match. 
For the CFHTLS-D4 field, the stellar samples consist of 
67 and 58 stars in the / and bands. The zeropoints of 
the HAWK-I images are adjusted to match the photome- 
try of these samples to the photometry of the same stars 
in the WIRDS data (see section 12.11 ). This process leaves 
residuals between 0.03 and 0.05 magnitude rms in the / 
and K s bands, respectively. The corresponding magnitudes 
are found to be in very good agreement with the photom- 
etry of the 19 2MASS stars present in the field, which is 
no surprise considering that the WIRDS data were cali- 



brated against 2MASS. We check that there are no sys- 
tematic offsets in the colours of our stellar samples com- 
pared to the c olours determined from the stellar library of 
iPicklesi (119981) and from a variety of stellar spectra mod- 
els at various temperatures and metallicites (iMarigo et al.l 
d2008l) and http : //stev . oapd. inaf . it/cgi-bin/cmd). 
For the GOODS-S field, we use the public ly available 
ISAAC J/H/K s catalogue (iRetzlaff et al.l (20101) to compute 
the zeropoint of our HAWK-I /-band image. For the Bullet 
Cluster field, the /-band and A^-band images are calibrated 
from the 2MASS catalogue, leaving residual errors of 0.05 
mag rms in both bands. 

For the calibration of the NB1060 data, because photomet- 
ric standards in narrow band filters do not exist, we perform 
the calibration direc tly on the image s tacks, following the 
approach detailed in lHibon et al.l d2010l) . It consists in inter- 
polating the NB1060 stellar photometry from the optical and 
NIR broad band data. This is justified by the large number of 
photometric datapoints available in at least two of our fields 
and by the absence of features at 1.06 fim in the infrared 
spectra of stars of spectral types earlier than M5 - the cold- 
est stars in our samples as determined by fitting their spectral 
energy distribution with the stellar models mentioned above. 
In practice the procedure consists in performing an ad hoc 
cubic spline fitting, for each star, of their magnitudes in all 
available bands. The magnitudes in the NB1060 band are de- 
rived from this fit. The procedure is adjusted according to the 
broad band data available in each field. We use exactly the 
same approach for calibrating the Y image in the case of the 
Bullet Cluster. 

For the CFHTLS-D4 field we use the u* g', r', V, z' optical 
data from the T0006 CFHTLS release and the NIR /, H, and 
K s WIRDS data mentioned above. The selection of stars that 
are neither too bright nor too faint in any of the available 
images leaves a sample of 23 objects. The residual error on 
the determination of the zeropoint from this sample is 0.05 
mag rms after rejection of outliers. 

For the GOODS-S field, the optical F435W, F606W, 
F775W, and F850LP magnitudes are taken from the 
merged HST/ACS catalogue (version r2.0z) available on the 
GOODS website. In this field, there are 44 suitable stars, and 
the process leaves a residual error of 0.06 mag rms in deter- 
mining the NB1060 zeropoint. 

For the Bullet Cluster we use a slightly modified procedure 
to calibrate the Y and NB1060 images. This is because of 
the lack of optical data for the entire field of view covered 
by HAWK-I, preventing us from performing a robust in- 
terpolation based on a large number of stars between the 
two wavelength ranges. Instead, we empirically determine 
the NB1060 zeropoints by matching the / vs / - NB1060 
(resp. K s vs NB1060 - K s ) colour-magnitude diagrams of 
the stars present in this field to the same diagrams produced 
on the GOODS-S and CFHTLS-D4 fields after calibration. 
The same procedure is used to calibrate the T-band image 
by comparing it to the colours of the stars in the GOODS-S 
field for which Y band data are available. 
All the procedures described above are conducted quadrant 
by quadrant, with a further iteration on the full four quad- 
rant images. In total, considering the consistency between 
the many checks that are performed, and despite the various 
methods used, we estimate that the final accuracy of the pho- 
tometric calibration is of the order of 0.1 magnitude rms. 
9. Absolute astrometric registration and final image stitch- 
ing. The last step in our reduction process consists in stitch- 
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ing and registering the four detector images to the reference 
images of each field. The final astrometric solution is com- 
puted by Scamps with a fourth-order polynomial fit of the 
star positions. The CFHTLS-D4 stacks are aligned to the 
archival CFHTLS images. The GOODS-S stacks are aligned 
to the optical HST/ACS images, and the Bullet Cluster stacks 
are aligned to the 2MASS catalogue in the absence of astro- 
metrically calibrated data across the entire area. The final 
astrometric residuals are below the 0705 rms level across 
the entire field of view for all images. Finally, the resam- 
pling and final im age stitching are performed using Swarp 
dBertin et al. 2002j). In addition, Swarp propagates the astro- 
metric solution to the weight maps and uses a weighted mean 
to compute pixel values in the small overlap between quad- 
rants due to the dithering pattern. 



rors computed from the pixel-to-pixel variance cr\ and the num- 
ber of pixels N ap in the aperture. The ratio of these two quanti- 
ties f corr = (AFl p /(N ap X cr\)) gives the noise-correction factor 
that can be used to correct the photometric errors measured by 
SExtractor. The ratio f co „ clearly depends on the aperture size: 
for apertures smaller than the correlation length of the noise, 
which is related to the size of the resampling interpolation ker- 
nels, fcorr ~ 1, whereas for large apertures f corr oc N pi \, ranging 
from 1.11 for a three-pixel diameter aperture to 1 .67 for a 25 
pixel diameter aperture. The procedure is repeated ten times for 
each field, and all three fields give similar and consistent mea- 
surements. 

This analysis finally allows us to assign signal-to-noise ra- 
tios (SNR) to the objects detected by SExtractor, in the sky back- 
ground limited regime, using 



3.3. Final image properties 

We now discuss the global properties of the final images: image 
quality (FWHM), noise, and detection limits. For consistency, 
the same procedure is applied to all the images used in this work, 
including archival data. The FWHM and the detection limits are 
listed in Table Q] 



SNR = 



fcorr AFse 



(1) 



where AF$e is the photometric error measured by SExtractor. 
The relation between SNR and the magnitude error Am is finally 
given by 



Am = 1.086 /SNR. 



(2) 



3.3.1. Image quality 

The image quality is determined from a high signal-to-noise ra- 
tio point spread function (PSF) generated by stacking unsatu- 
rated and isolated stellar images (range of 20-30) in each field. 
After normalization, the stellar images are centred and median- 
stacked with a 4<x outlier rejection. The rejection reduces the 
contribution from faint neighbouring objects in the wings of the 
PSF but does not affect its profile. The resulting NB1060 images 
are slightly elongated, for reasons that are unknown to us, with 
a measured ellipticity from about 0.05 to 0.1 along a direction 
< 10° away from the N-S axis. The FWHM values are derived 
from a 2D-Gaussian fit to the median profile. The three NB1060 
final images have exquisite image qualities ranging from 0753 to 
0758 (see Table[B. 

3.3.2. Photometric errors and correlated noise 

Image resampling introduced by the distortion correction, shift- 
ing, stitching, and registration processes introduces correlation 
in the noise of the images. This leads in turn to underestimat- 
ing the photometric erro rs when considering the pixel-to-pixel 
noise properties , see e .g. iGrazian et"aD (120061) . Appendix A, or 
ICasertano et al.l (120001) . To measure and account for this well 
known effect, we carefully analyse the noise properties of the 
images over apertures of varying sizes and derive correctio n fac- 
tors that we can then apply to the photometric data measured 
by SExtractor. Indeed, SExtractor derives the photometric er- 
ror for each object it finds by computing the local pixel-to-pixel 
noise fluctuation in the vicinity of the object (in the faint-object, 
background-limited regime). The SExtractor photometric errors 
are therefore affected by the correlation of the noise. For each 
image, we select a thousand positions corresponding to source- 
free background regions determined from the final SExtractor 
segmentation (object mask) image. For each position, we mea- 
sure the integrated flux in circular apertures of diameters rang- 
ing from N ap = 1 to 25 pixels (07 1065 to 27663). For a given 
aperture size, the variance of these fluxes, AF% p , differs, because 
of the correlated noise, from the variance N ap X cr\ of the er- 



3.3.3. Aperture corrections and optimal apertures 

We measured curves of growth on unsaturated stars for all im- 
age stacks used in this work, using apertures between 1 and 150 
pixels (071065 to 16") in diameter. Less than 1% of the flux 
resides in the wings of the PSF beyond radii of 7", and we 
therefore safely use the 16"aperture correction to estimate the 
total flux of unresolved or moderately resolved objects. From 
the curves of growth of both flux and noise, we derived the opti- 
mal diameter that maximizes the signal-to-noise ratio for point- 
like objects. In practice, for all of our NB1060 image stacks, 
a diameter of Q) — 0764 (6 pixels) is used. The correspond- 
ing aperture corrections 5m ap for the three NB1060 final im- 
ages are 6m ap = 0.90 + 0.04 mag, 5m ap = 0.82 + 0.03 mag, 
5m ap = 0.85 + 0.04 mag for the GOODS-S, CFHTLS -D4, and 
Bullet Cluster fields, respectively. The corresponding noise cor- 
recting factors are f corr = 1 .22, 1.14, and 1 . 14 for the GOODS-S, 
CFHTLS -D4, and Bullet Cluster fields, respectively. 

3.3.4. Detection limits 

Finally, from the two parameters defined above, f corr and 6m ap , 
one can define the Icr limiting magnitude m\o- for point-like ob- 
jects: 

mio- = -2.5 log 10 (/ corr AF S£ ) - Sm ap + ZP, (3) 

where f cor r, AF$e and 6m ap correspond to apertures of 0764 in 
diameter. This leads to a 3crNBl 060 point source detection limit 
of m 3o - = 26.65, 26.65, and 26.50 for the GOODS-S, CFHTLS - 
D4, and Bullet Cluster, respectively, as reported in Table [1] 

4. Candidate selection 

4. 1 . Detection completeness 

We use SExtractor dBertin & Arnoutslll996l) version 2.8.6 for 
source detection and photometric measurements. We fit the PSF 
median profile discussed in section I3.3.1I with a sum of three 
2-dimensional Gaussians from which we derive the filter (9x9 
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Fig. 3. Completeness levels of the NB1060 images for the three 
observed fields. The coloured areas correspond to the Poissonian 
errors on the number counts in the corresponding magnitude bin. 
The triangles indicate the 5cr magnitude limits. 



pixels or 0796 x0796) used by SExtractor for spatial filtering dur- 
ing the detection process. We then use this point-source model 
to generate mock sources injected into the image to estimate the 
image detection completeness. We inject 500 mock sources per 
Am = 0.1 magnitude bins in regions of the images randomly 
distributed and free of objects. We perform a number of tests 
to determine the optimum SExtractor detection parameters that 
maximize the number of detected objects while minimizing the 
number of false alarms. Considering the absence of candidate 
LAEs in our images, we choose to push to the faintest possi- 
ble limits. False alarms are investigated by running SExtractor 
on the negative images and are discussed in section 1431 We de- 
termine that an adequate set of SExtractor parameters, for the 
purpose of our analysis, is to trigger a detection on one pixel 
(DETECTJVIINAREA) after spatial filtering at 0.7cr above the 
local background (DETECT.THRESH). With these detection 
parameters, the average signal-to-noise ratio of sources in the 
50% completeness magnitude bin is of about 4, and a SNR of 5 
corresponds to a completeness rate of 70 to 80%. Figure [3]shows 
the completeness rates achieved in the three NB1060 images. 

4.2. Selection criteria 

We do not expect z ~ 7.7 LAEs to be detected in any of the 
filters blueward of the Lya line redshifted to 1.06 /urn. First, neg- 
ligible amounts of radiation are expected to escape the galaxy 
and to be transmitted by the IGM below the Lyman limit, which 
is redshifted to ~ 790 nm. In addition, all the radiation between 
the Lya and Lyy lines at z = 7.7 is entirely redshifted beyond 
the Gunn-Peterson trou gh at ~850 nm observed in the spectra 
of high-redshift quasars (IFan et al.ll2006l) . Only in the blue part - 
the most depressed part - of the Lya forest, just above the Lyman 
limit, can we therefore expect some flux from a z ~ 7.7 LAE to 
arrive on Earth, in the wavelength range [790-850] nm approx- 



imately. In practice, considering the limiting magnitudes of our 
optical and NB1060 images, an absorption of 2 magnitudes or so 
will result in no detections in any of the optical bands. 

For the purpose of the analysis described in this paper, we 
built a master catalogue of all the NB1060 detected objects, mea- 
suring their magnitudes in each of the optical and NIR broad 
band images by running SExtractor in double image mode. To 
do so, we resample all images to the HAWK-I pixel scale of 
O/1065. We then search objects in this master catalogue that 
are not detected in the optical images at an initial SNR < 3 
level, reduced to SNR < 2 after visual inspection. In the case 
of the CFHTLS-D4 field, we request in addition that these ob- 
jects not be detected at a 2<x significance level in a x 1 image 
of the field obtained by combining the g', r', and V images. In 
the case of the GOODS-S field, we use a similar non-detection 
limit on a bviz x 2 image obtained by combining the four broad 
band HST/ACS images. Because the NB1060 bandpass is lo- 
cated within the bandpass of the Y filter (at its red edge), the Lya 
line may be detected in the Y filter. To estimate the Y— NB1060 
colour as a function of redshift, we generate simple synthetic 
models of LAE spectra, consisting of a narrow Lya line and a 
UV-continuum of energy distribution fx oc We allow the UV 
slope B to vary from -3 to 0, and we set the flux density below 
the Lya line to zero. For objects with redshifts in the interval 
[7.70 - 7.78] corresponding to the NB1060 filter, we find that 
the Y- NB1060 colour is ~ 2.35 ± 0.35, with the lowest values 
corresponding to situations where the Lya line is redshifted near 
the edges of the NB1060 filter transmission curve, and therefore 
strongly attenuated. This corresponds to the case of high-redshift 
LBGs detected through their continuum in the NB1060 filter. 
As we see in the next section, the Y images (when available) 
are not deep enough, relative to the NB1060 images, to measure 
such a colour on the faintest objects, but they do allow us, con- 
versely, to discard blue and moderately bright objects that pass 
the other colour selection criteria. To secure the presence of an 
emission line in the NB1060 filter, we further require a lcr NB 
ex cess over the flux m easured in the /-band. From equation (6) 
in lHibon et al.l d2010h . NB1060 - J < corresponds to equiva- 
lent widths £W bs > 50Aor£W res t > 5. 7 A, assuming a flat con- 
tinuum spectrum (f v = const). We note that the NB1060 filter is 
placed approximately at the centre of the bandpass covering the 
Y and J filters, allowing us to further constrain the presence of an 
LAE from its colour between the NB1060 and Y+J bandpasses; 
however, in the absence of candidates from the criteria used so 
far (see next section), this did not prove necessary to add to our 
selection criteria. Finally, we restrict the analysis to sources hav- 
ing SNR > 5 in the NB1060 final images and SNR > 2 in partial 
or intermediate image stacks corresponding to different observ- 
ing epochs. In summary, our detection criteria are 

1. NB1060 > 5o~ A NB1060 epoM > 2<x A NB1060 epoch2 > 2cr, 

2. no detection above the 2cr level in any of the visible broad 
band filters, 

3. 2 < Y - NB1060 < 2.7 (when Y band data are available), 

4. NB1060 - J < with lcr significance. 

We finally note that astrophysical sources such as transients, 
extremely red objects (EROS), high-EW low-z line emitters or 
T-dwarfs can potentially satisfy the optical non-detection crite- 
ria defined here. Relatively deep NIR Y and/or J and/or K s band 
data are therefore required for consolidating the selection (e.g. 
criteria #3 and #4) and reducing contamination from astrophys- 
ical sources. The J and K s band data in particular are useful to 
identify T-dwarfs and EROs, even if the latter are often detected 
in deep optical images. In the absence of LAE candidate in our 
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data, we are clearly not affected by contamination, thanks to 
the coherent data sets that we use. We refer the interested reader 
to section 3.3 of iHibon et alj d2010t) for a somewhat more de- 
tailed analysis of contamination effects in a similar z-11 LAE 
search, in particular by Ha, [OIII], and [Oil] line emitters. 

4.3. Selection field by field 

The 5<x detection limit in the NB1060 filter corresponds to mag- 
nitudes m^u- = 25.9 to info = 26.1 depending on field (see 
Table[T]i, and to a completeness level of 70 to 80% (see figure[3]l. 
The corresponding colour criteria used for the selection of can- 
didates differ among the three fields depending on the depth of 
the optical images available in each of them. 

CFHTLS-D4. The selection criterion #2 in the previous sec- 
tion corresponds to the following colour criteria: 

u* 2cT -NB1060 > 1.7, 

g'^-NB1060 > 2.5, 

rV - NB1060 > 2.3, (4) 

i' 2a -NB1060 > 1.8, 

z' llT -NB1060 > 0.9. 

There are ~ 6500 NB1060 objects detected in this field. The 
application of criteria #1 and #2 of section 14.21 yields 20 ob- 
jects. Ten are visually identified as instrumental artefacts caused 
by electronic crosstalk (see section 14.41) . Seven of the remain- 
ing objects are detected and relatively bright in the / band and 
therefore rejected after application of criterion #4. We note in 
passing that the brightest of these objects has NB1060 = 24.15, 
J = 23.85 and J - H 2lT < -0.85 and is very likely a T-dwarf. 

Amongst the three remaining objects, one is located near the 
edges of the image and appears sharper than the PSF. One is lo- 
cated in the wings (4") of a bright extended galaxy, therefore of 
suspicious photometry and therefore unusable. Finally, the last 
one appears to be a variable, extended object, with > 1 mag- 
nitude difference between the first- and second-epoch observa- 
tions. All three objects are therefore discarded. 

To allow for possibly slightly extended LAE candidates and 
for consistency checks, we carry out a second selection using 
larger apertures (© = 1//065 = 10 pixels ~ 2 x FWHM), and 
applying the appropriate aperture correction. Criteria #1 and #2 
yielded 14 detections, out of which nine are in common with the 
previous sample of 20 objects. Amongst the remaining five new 
detections, one is an obvious artefact near a bright star, two are 
detected in the /-band and are rejected due to the low signifi- 
cance of their NB excess. The two last ones are low significance 
detections in the NB1060 image (SNR < 5.5) and both show 
extended and dubious morphologies. 

Checking the robustness of the rejections further, we 
investigate the false alarms on the negative image using the 
two aperture diameters mentioned above. After removing the 
well-determined crosstalk features, which have a negative 
component, we are left with eight objects using the ® = 0764 
aperture and 2 using the Q) = 17065 aperture, all of them at the 
limit of our signal-to-noise ratio selection. These genuine noise 
artefacts have very similar morphologies to those of the positive 
detections that were rejected on the science image: either very 
sharp or extended and irregular with bright non-contiguous 
pixels. This legitimizes our somewhat ad hoc, but pragmatic 
earlier selection based on the morphology of the faintest positive 
candidates, and we therefore conclude that there are no z = 7.7 
LAE candidate in the CFHTLS-D4 field. 



GOODS-S. Here the selection criterion #2 corresponds to 
the following colour criteria: 

F435W 2a -NB1060 > 2.3, 
F606W 2cr - NB1060 > 2.5, 

F775W 2lr -NB1060 > 1.9, (5) 
F850LP 2a - NB1060 > 1.6. 

There are ~ 5100 NB1060 detected objects. The applica- 
tion of criteria #1 and #2 yields 16 objects, of which 12 are vi- 
sually identified as instrumental artefacts caused by electronic 
crosstalk. Amongst the four remaining objects, one is due to a 
mismatch on a blended object, and two are marginally detected 
objects at the edges of the image. The last object is detected 
in the 7-band but does not satisfy criterion #3 above; interest- 
ingly, it is identified wi th reference G2_1408 as a z ~ 7 object in 
ICastellano et al.l (1201 Oi and refere nces therein). This o bject has 
been followed up in spectroscopy dFontana et al.ll20l"oh . yielding 
a tentative detection of the Lya line at a redshift z = 6.97. This 
object therefore appears to be an LBG, caught by its strong UV 
continuum emission detected in the NB1060 filter. 

Similar to what was done on the CFHTLS-D4 field, we then 
performed a second selection using Q) = 2 x FWHM = 17065 
apertures. This yields two new detections (beyond the obvious 
electronic artefacts): one is detected in the F-band and does not 
pass criterion #3, and is also marginally noticeable in the optical 
bands. The other one has a dubious morphology and is close to 
a bright object, and is therefore rejected. Finally, we carried out 
the false alarm analysis on the negative image, and we detect 
a handful of events with dubious morphologies, leading to the 
same conclusions as for the CFHTLS-D4 field. 

Bullet Cluster. The dataset for the Bullet Cluster field is 
somewhat different than for the other fields. As explained in sec- 
tion 12.31 we accomodated a consistent set of HAWK-I data in 
the Y, NB1060, J and K s bands within our large programme. In 
addition, we used HST/ACS images of the inner part of the field, 
as well as moderately deep IM ACS images from the Magellan 
telescope dClowe et al]l2006l) . There are ~ 7000 NB1060 ob- 
jects satisfying criterion #1, the vast majority of which are de- 
tected in the Y filter and do not satisfy criterion #3. Only 127 
objects are not detected in the Y image at 3<x, but because 
Y31T - NB1060s lT = 0.4, it is impossible to conclude whether 
they satisfy criterion #3 or not. After visual inspection and re- 
jection of electronical ghosts and obvious artefacts, all but one 
object show a clear counterpart in either one of the HST/ACS 
images or in the IMACS images. This object shows no NB ex- 
cess (NB1060 - J - 0.55 + 0.25), and together with a marginal 
detection in the 7-band (SNR ~ 2.4, Y = 26.40 + 0.45) and a 
non-detection in K s (K s < 25.3, 3<x upper limit), we conjecture 
that this object is probably a T-dwarf. Finally, a selection based 
on larger apertures as for the two other fields yields no new can- 
didates. We therefore conclude, again, that there are no z — 7.7 
LAE candidates in the Bullet Cluster field. 

4.4. Instrumental artefacts 

Instrumental artefacts are a potentially important source of con- 
tamination. As explained in section 14.31 candidates are found 
that are rejected as instrumental artefacts. We describe here some 
of the instrumental artefact sources that are observed in the 
HAWK-I data. 

1. Electronic crosstalk. The HAWK-I data suffer from inter- 
channel crosstalk from the readout electronics. This results 
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in donut-shaped artefacts at regularly spaced intervals along 
detector lines where bright stars are present. These artefacts 
were largely attenuated after a technical intervention in the 
instrument that took place in May 2009. The crosstalk pat- 
tern follows the dithering pattern and is therefore present on 
the final stacked images. The crosstalk artefacts are easy to 
recognize from their shapes and fixed distances from bright 
stars along detector rows. Because they do not have counter- 
parts in optical images, these artefacts are selected as candi- 
dates in our analysis, but are easily dealt with a posteriori. 
No attempt was made to remove these artefacts during data 
processing. 

2. Optical ghosts. Reflections inside the instrument gener- 
ate typical out-of-focus and decentred pupil images around 
bright stars. The surface brightness of these haloes was mea- 
sured to be 10~ 4 of the peak intensity in the PSF profiles. 
Only focussed optical ghosts can be mistaken as candidates, 
and no such artefacts are observed on the HAWK-I images. 

3. Persistence. Persistence from previously observed bright 
stars is at a fixed detector position. The persistence features 
therefore do not follow the dithering pattern and are rejected 
when combining the images with sigma clipping in the fi- 
nal stacks. Considering the large number of frames used in 
the stacks (more than 200 frames), persistence effects are un- 
likely to leave residuals that can be mistaken as candidates. 

4. Radioactive events. One of the HAWK-I arrays (chip #2, Id: 
ESO-Hawaii2RG-chip78) suffers from a stron g radioactive 
event rate, coming from the detector substrate dFinger et alJ 
l2008h . These radioactive events generate showers of variable 
intensity (typically thousands of electrons) and extent (typ- 
ically a few tens of pixels on a side). Some events can be 
as bright as a few hundred thousand electrons and extend 
up to 400 pixels in one direction. Because of the long de- 
tector integration times (DIT) used for the NB1060 images 
(300 seconds), there are a few tens of such events in a sin- 
gle frame. This results in poor background subtraction and 
moderately high-frequency residuals (a few tens of pixels) in 
individual sky-subtracted frames. Although the global noise 
properties are not significantly different in this quadrant than 
in the others, the overall cosmetics are somewhat poorer. As 
a consequence, most of the low signal-to-noise ratio detec- 
tions and with dubious morphologies reported in section 14.31 
appear to be in this quadrant and are therefore rejected as 
artefacts after visual inspection. 

5. Noise. As explained earlier, we chose a low detection thresh- 
old in order to push to the faintest detection limits, triggering 
a handful of low signal-to-noise ratio false alarms, particu- 
larly on the highly radioactive quadrant. We settle on a de- 
tection threshold that allows us to handle these false alarms. 

5. Constraints on the z - 7.7 Lye* luminosity 
function 

5. 1 . Comoving volume 

The effective field of view in the transverse dimension of the 
NB1060 stacks is computed as a function of the detection limit 
for each field from the background noise maps generated by 
SExtractor, and accounting for the correlation of the noise as de- 
scribed in section [3.3. 21 The sensitivity is reduced at the edges 
of each quadrant image thanks to the dithering process and in 
regions close to bright objects. All objects are masked, reduc- 
ing the total effective area by ~ 10% for the blank fields, and 
~ 25% for the Bullet Cluster due to the large number of bright 



galaxy cluster members. For this field, we further use a map of 
the gravitational amplification and space distortion from a de- 
tailed model of the cluster (Richard et al. in preparation) to com- 
pute the eff ective, unlensed, c omoving area corresponding to the 
image (see IWillis et al.l |2008[ for a similar example). In the di- 
rection of the line of sight, we use the filter transmission curve to 
determine the effective width. We then approximate the NB1060 
filter with a rectangular filter of width equal to this effective 
width. The effects of the filter transmission cu rve on LAE detec- 
tion an d co moving volumes are detailed in e.g. IWiUis & CourbinI 
d2005b and lHu et al.1 J20!ob . Considering our null results, these 
effects will not affect our conclusions. We also assume that the 
filter transmission curve does not vary significantly over the in- 
strument's field of view, an assumption motivated by the relative 
uniformity of the sky background over the instrument's field of 
view. We finally assume that the observed Lya emission lines 
are significantly narrower than the filter width, as is the case for 
high-z LAEs, and we therefore ignore the effect of the line width 
on the line flux measured through the NB1060 filter. The effec- 
tive width of the NB1060(AA eff ~ 100A) defines the [7.70-7.78] 
redshift interval probed by our observations. 

Converting magnitudes to line luminosities requires assump- 
tions on the equivalent width (EW) of the Lya line. Distributions 
of observed Lya lines in high-z LAEs vary from a few tens of 
Angstroms to lower Umits of a few hundr ed An gstroms (see e.g. 
iTaniguchi et al.l 120051: lOuchi et all 1201 Oh . The ITaniguchi et all 
(120051) distribution of Lyo- line EWs is consi s tent w ith a con- 
version factor of 70%, as used in IHibon et al.l (1201 Ob . when as- 
suming that the EW low er limits are the real values. Conversely, 
iKobavashi et alJ ( 1201 Oh predict a distribution of Lya EWs 
clearly peaked toward high values, hence favouring conversion 
factors closer to 100%. In the following, we therefore use these 
two values (70% and 100%) when converting NB1060 magni- 
tudes to line luminosities. 

In total, the comoving volume sampled by our images is 
5.9 x 10 3 Mpc 3 for the Bullet Cluster and 9 x 10 3 Mpc 3 for 
each of the two blank fields, corresponding to a grand total of 
~ 2.4 x 10 4 Mpc 3 for the three fields. The volume-luminosity 
relation is shown in figure |4] and the effect of the gravitational 
amplification for the Bullet Cluster is clearly visible, where the 
gravitational amplification enables probing fainter luminosities, 
however over increasingly smaller volumes. 

5.2. Cosmic variance and Poisson noise 

We do not detect any LAE candidates down to a NB1060 5<x 
limiting magnitude of ~25.9 to ~26.1. To constrain the luminos- 
ity function of z = 7.7 Lyo- emitters and compare our results 
with o thers, we make use of the Schechter formalism (Sc hechterl 
fl976h : 

(L)dL^*(Afexp(-A) d (il) (6) 

where L* is the characteristic luminosity defining the LF high lu- 
minosity cutoff, <p* a volume density normalization factor, and a 
the faint-end slope characterizing how steeply the LF increases 
at low luminosities. Limited samples and large errors lead to 
degneracy between these three LF parameters. Pending more ob- 
servational data and more accurate determinations of the high-z 
LAE LF parameters, most of the authors in the literature have 
settled on a canonical faint end slope value of a = -1.5. Our 
observations probe luminosities similar, or slightly fainter, than 
those observed by other groups; therefore, we can compare our 
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Fig. 4. The comoving volume V(L) sampled by the NB1060 im- 
ages as a function of Lya luminosity. The two dashed curves 
correspond to the two blank fields (green - CFHTLS-D4; red 
- GOODS-S). The thin black curve corresponds to the Bullet 
Cluster. The thick black curve is the total comoving volume cor- 
responding to the three fields. The Lya luminosity corresponds 
to a 5cr NB1060 limiting magnitude, assuming a 70% average 
conversion factor between the Lya luminosities and NB1060 
magnitudes (see text). 



results to others, and to this aim we similarly adopt, unless stated 
otherwise, a faint end slope of a — -1.5. 

Denoting V(L) the comoving volume probed by our observa- 
tions as a function of the luminosity L, as described in section lBTTI 
and shown in figure[4] the total number of objects N{L* , <p*, a) is 
given by 



J 



N(L*,<f>*,a) = <f>(L)V(L)CF(L)dL 



(7) 



where CF(L) is the completeness function (see section 147Tb . The 
conversion of the completeness function as a function of magni- 
tude as shown in figure[3]to CF(L) takes into account the conver- 
sion factor of 70% and 100% mentioned above, and in the case 
of the Bullet Cluster further takes the amplification map of the 
cluster into account. 

From a Poisson distribution, one can easily compute single- 
sided confidence levels (CL) for the upper limits of the expected 
number of e vents TV,, that c orrespond to a measured number of 
events n, as dGehrelsll986T) : 



A=0 



-N. II* _ l _ C T 



(8) 



In our situation of zero detection (« = 0), the 84.13 , 97.72, 
99.87, and 99.99 percentiles 5 confidence levels correspond to 
upper limits of the mean number of events N„ of 1.84, 3.78, 



5 Corresponding to the single sided l<x,2cr,3<x,4cr percentiles of a 
normal distribution 



6.61, and 10.36, respectively. The situation N u < 1 corresponds 
to a 63% confidence level. Therefore, with zero detection and as- 
suming pure Poisson statistics, one can exclude, at a given con- 
fidence level CL, the luminosity function parameters that would 
yield an expected number of objects N u with our survey param- 
eters. 

However, considering the somewhat limited area covered 
by our observations, we need to co nsider the effects of cos mic 
variance in our statistical analysis. ISomerville et al.l (120041) are 
among the first authors to derive quantitative estimates of the ef- 
fects of cosmic variance from cold dark matter (CDM) models 
and observations of the two-point correlation f unctions of galaxy 
popula tions in the GOODS survey data. iTrenti & Stiavellil 
(120081) expand on this work and propose a cosmic variance 
model, based on N-body simulations and halo occupation distri- 
bution models, and applied to a variety of high-z galaxy popula- 
tions. We use below the on-line version of this model to estimate 
the effects of cosmic variance on our observations. 

Various prescriptions have been proposed for the distribution 
fun ction of galaxy number counts affected by cosmic variance, 
see lYang & Saslawl d201 lb for a recent discussion and analysis 
of the galaxy counts-in-cells distribution functions in the SDSS 
data. Although not physically motivated, the negative binomial 
distribution (NBD) fits the SDSS data well at both low and high 
number counts and provides a convenient description for the dis- 
tribution function of galaxies with positive number counts and 
overdispersion relative to a Poisson distribution. To prevent tech- 
nical problems with the use of normal or lognormal distributions 
(e.g. truncation to positive numbers), we chose to adopt the NBD 
as an ad hoc representation of the probability density function of 
low galaxy number counts. The NBD can be conveniently ex- 
pressed as a Poisson random variable whose mean population 
parameter is itself random and distributed as a Gamma distribu- 
tion of variance equal to the relative cosmic variance. 

By definition of the cosmic variance, the variance N 2 - N 
of the number of galaxies of mean number N is in excess of the 
Poisson variance N and is given by 



N 2 -N = /V xcrl+N 



(9) 



where cr 2 is the relative cosmic variance. With the NBD prescrip- 
tion, one can derive the confidence level CL corresponding to no 
detections in our observations, for a known cr 2 . To estimate the 
cosmic variance, we run the on-line cosmic variance calculator 6 
(ITrenti & Stiavellill2008l) and derive cr 2 for low number counts 
N. We use the parameters corresponding to our observations: the 
instrument's field of view, a mean redshift of 7.74, and a redshift 
interval of 0.08. We select the Sheth-Tormen bias formalism, a 
value of 0.8 for cr g , and a value of 1.0 for the user-selectable 
halo-filling factor. Although the filling fact or of LAEs i s likely 
to be significantly lower than 1.0, see e.g. lOuchi et al.l d2010l) . 
adopting a value of 1 .0 leads to overestimating the importance of 
cosmic variance and therefore corresponds to a worst-case sce- 
nario in our analysis. Finally, we define a completeness factor 
of 1 .0 in the cosmic variance calculator since we use our mea- 
sured completeness function in equation|7]to derive the number 
of galaxies that correspond to a given set of LF parameters. We 
then fit the resulting relative cosmic variance cr 2 corresponding 
to our survey parameters as a function of N, and we plug this cos- 
mic variance into a Gamma distribution. For each pair of (L*, <p*) 
parameters we thus derive a confidence level corresponding to 
no detections in our observations. 



http://casa.colorado.edu/~trenti/CosmicVariance.html 
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This is represented in figure [5] together with the best-fit pa- 
rameters of the Lya LAE luminosity functions at redshifts 5.7 
and 6.5 from various sources. We report in this plot the range of 
(L*, <p*) parameters excluded at 85% and 99% confidence levels, 
for conversion factors between NB1060 magnitudes and Lya lu- 
minosities varying from 70% and 100% (see discussion in sec- 
tion El). We report the upper (L*,<p*) exclusion zones derived 
from the absence of detections in the HAWK-I observations only 
(this work). Furthermore, we report the upper exclusion zones at 
the same confidence levels when adding to the null HAWK-I 
detections the null spectroscopic confirmati on of the five bright- 
est z = 7.7 LAE candidates 7 presented in iHibon et"aT1 (120101) . 
Finally, for illustration, we also report the 85% confidence level 
derived from this work (HAWK-I observations only) for a faint 
end slope of the z = 7.7 Lya LAE LF a = -1 .7. 



6. Discussion 

The various LF parameters reported in the literature at redshifts 
5.7 and 6.5 present some significant differences. We report in fig- 
ure [3] some of the best-fit Schechter parameters at redshifts 3.1, 
5.7 and 6.5. We report photometric samples partially confirmed 
in spectroscopy and purely spectroscopic samples. At the same 
redshift, the two series of datapoints clearly differ by significant 
amounts. 

lOuchi et al.1 (1201 Oh. and previou sly iKashikawa et alJ (120061) 
followed bv lKashikawa et aTJd2Ql ll) . convincingly claim that the 
evolution, mostly in luminosity, of the z ~ 6.5 LF from the lower 
redshift LFs at z ~ 3.1 and z ~ 5.7 LFs is a signature of reion- 
ization, due to a neutral-hydro gen fraction xu i of the order of 
20% at z = 6.5. Conversely, the lHu et ail d2010l) evolution of the 
LF parameters between the two same redshifts is mostly in den- 
sity; accordingly, they do not infer a signature of reioniza t ion, in 
agreement with an earlier claim by iMalhotra & Rhoadsl (120041) 
that there was no evidence of reionization in the evolution of the 
LAE LF between these two redshifts. 

In the light of the recent work from IHu et al.l (1201 Oh . the 
question of the evolution of the Lya LAE luminosity function 
beyond redshift 5.7 becomes more d ifficu lt to comprehend. If 
the z = 5.7 point of IHu et alJ (1201 Ol) (and IMalhotra & Rhoadsl 
120041) is correct, the picture by which the LAE LF would be ap- 
proximately constant between z = 3 and z = 5.7, and would 
evolve significantly beyond redshift ~ 6 as a consequence of an 
increasingly neutral intergalactic medium (IGM), might need to 
be revisited. How useful are our results in this context? 

From figure|5] we infer that we can safely exclude a non evo- 
lution of the LF par ameters at z — 7.7 at more tha n 99% confi- 
dence level from the IKashikawa et aD (I2006L l201lh . lOuchi etalJ 
(120081) and lOuchi et al.l (l2010h values at z = 5.7 and 6.5. In 
that case, we should find ~ 13.7 and ~ 11.7 LAEs in the 
three HAWK-I fiel ds b ased on the LF paramete rs estimated by 
lOuchi et all d2Q10h and IKashikawa et al] fcOl lb at z = 6.5, re- 
spectively. The LF parameters of Hu et al. (1201 Ol) would pre- 
dict ~ 2.5 LAEs at z = 6.5 and can be similarly excluded at 
an ~ 85% confidence level, or higher, when including the null 
spectroscop i c confi rmation of the brightest LAE candidates of 
IHibon et"a"fl d2010l) (see previous section and footnote [7J. Our 
results therefore clearly show that the z = 7.7 Lya LAE LF does 
evolve from lower redshifts, but in the absence of concordance 
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Fig. 5. Parameters of the z = 7.7 luminosity function excluded 
at 85% and 99% confidence levels from our data, assuming 
the Schechter formalism and a fixed faint end slope a = -1.5 
unless otherwise stated. Filled circles correspond to the best 
fit LF parameters at z = 5.7 (blue) and z = 6.5 (red) from 
IKashikawa et alJ ( 1201 lh . The ellipses correspond to the 3cr confi- 
dence levels for these datapoints. Open circles corre spond to the 



best fi t LF parameters of the photometric samples of Ouchi et al 
( 2008 ) at z — 3 . 1 (magenta) and z — 5 .7 (blue) and of lOuchi et al 



2010) at z - 6.5 (red). The ellipses correspond to the 2cr conli- 



7 Indeed, spectroscopic follow-up with X-shooter at the VLT of the 
five brightest candidates failed to detect Lya emission. These results 
will be reported separately in Cuby et al. (in prep.). 



dence levels for these datapoints. The filled square symbols cor- 
respond to the best fi t LAE LF param eters at z = 5.7 (blue) 
and z = 6.5 (red) from lHu etaTl (120101) . The plain (resp. dotted) 
black lines correspond to the 85% and 99% confidence levels 
corresponding to no detections in our HAWK-I data (this work), 
assuming a conversion factor of 70% (resp. 100%) between the 
NB1060 and Lya fluxes. The dark (resp. light) grey zones cor- 
respond to the range of 85% (resp. 99%) confidence levels for 
conversion factors between 70% and 100%, as delimited by the 
plain and dotted lines. Similarly, the lines and hatched zones in 
magenta colour correspond to the 85% and 99% confidence lev- 
els when adding to our work the null spectroscop ic confirma- 
tion of the five brightest z — 7.7 LAE candidates of IHibon et al.l 
(120101) . 



between the data at lower redshifts, it is difficult to ascribe this 
evolution to the galaxy properties or to reionization. 

To continue the discussion, we now consider two scenarios 
for the evolution of the LAE LF between z = 6.5 and z = 7.7, 
either in density or in luminosity. These scenarios are purely 
phenomenological and are not supported by theoretical consid- 
erations, only serving the purpose of assessing what the conse- 
quences of our results might be. This is illustrated in figure [6] 
in a cumulative luminosity function plot. For a luminosity func- 
tion to be consistent with our null results, it should lie, ignoring 
statistical fluctuations, below the contours corresponding to the 
parameter space probed by our observations. 

In the first scenario, we consi der a ~ 60% ev olution in den- 
sity from the z = 6.5 datapoint of IHu et all d2010h . Such an evo- 
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lution can be entirely ascribed, in principle, to an intrinsic evolu- 
ti on of the LAE LF betwee n z = 6.5 and z = 7.7 as in the models 
of iKobavashi e t al. (2007]) where the z = 6.5 LF, multipling <p* 
by 0.4, almost perfectly coincides with the z — 7.7 LF. In prac- 
tice, such an evolution could result from the combined evolution 
of the density of dark matter haloes and of the intrinsic galaxy 
properties. We also note that the UV LF of high-redshift galax- 
ies between z ~ 4 an d z ~ 8 evolves essentially in luminosity 
dBouwens et al. I l201lh . so a pure evolution in density should be 
treated with caution. As can be seen in Figs. [5] and [6] such an 
evolution would fit most of the observation al data, the shap es 
of the lower-redshift LFs, the datapoint from Ive et al.l (120061) at 
z = 6.96, the datapoints from lVanzella et al.l (1201 lh at z = 7.008 



and z — 7.1 09, and our da t a. The mere existence of the live et al 



2006) and lVanzella et all d2011l) datapoints and of the lOta et a! 



(120101) LAE candidates at z ~ 7 provides interesting constraints 



at high luminosities that cannot be fully captured in the (L*, <p*) 
parameter space of figure [5] and are better seen in figure The 
main conclusion to be drawn from this test case is that to be con- 
sistent with our results, it does not require invoking a change in 
the Lya IGM transmission and therefore a change in the neutral- 
hydrogen fraction of the IGM. 

In the second scenario, conversely, we cons ider a 60% 
chang e in luminosity from the z = 6.5 LF of lOuchi et"al] 
(l2010h . This scenario clearly requires, by construction, a signifi- 
cant quenching of the IGM Lya transmission. The present lumi- 
nosity evolution sce nario may be compatible with the claim by 
iFontana et aHd2010l) of a significant quenching of Lya emission 
among LBGs at z ~ 7. Although strongly model dependent, a 
high neutral-hydrogen fraction of the IGM due to a still incom- 
plete reionization would then be req uired in this s cenario, e.g. 
Xm ~ 60% according to the model of lSantosI d2004l) . 

To conclude this discussion, we cannot safely decide on 
whether we are 'seeing' signatures of reionization in our results, 
as this depends on which assumptions we use for the LFs at 
lower redshifts. We call for clarifications of the LAE luminos- 
ity functions at redshift 6.5 and below, a prerequisite to draw- 
ing firmer conclusions on reionization as inferred from LAE ob- 
servations. We also note that our results are in cons istent with 
the ph otometric samples from lTiivi et al.1 (1201 Ol) and lKrug et al] 
d201 lh at z — 7.7, but spectroscopic confirmation is required be- 
fore drawing firmer conclusions. In the meantime, observations 
at z = 7.7 continue. An ideal complement to the HAWK-I obser- 
vations presented in this paper would be to probe higher lumi- 
nosities (of the order of 10 43 ergs.s -1 ) over significantly larger 
comoving volumes (a few times 10 6 Mpc 3 ). 



7. Conclusion 

We searched for Lya emitters in three fields, two blank fields 
and one cluster field with the HAWK-I instrument at the VLT, 
using a NB filter centred at 1 .06 //m. Our data in this filter to- 
tal 80 hrs of integration time. The total comoving volume is 
~ 2.4 x 10 4 Mpc 3 . We reached a 5cr limiting magnitude that we 
use as detection threshold, of ~ 26.0+0.1 AB magnitude. We se- 
lected the objects from various colour criteria, which depend on 
the auxiliary data available for each field. We did not detect any 
object matching our selection criteria that would correspond to 
z = 7.7 LAEs. We modelled the probability density function of 
high-z LAE populations including Poisson statistics and cosmic 
variance with a negative binomial distribution. From this statisti- 
cal description and from the absence of LAE detections, we ex- 
clude a non-evolution scena rio at ~ 85% (resp . 99% ) confidence 
level from the z = 6.5 LF of iHu et all d2010l) (resp. lOuchi et al.l 
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Fig. 6. Cumulative Lya luminosity functions. The blue (resp. 
red) lines show the c umulative luminosi ty functions at z = 5.7 
(resp. z - 6.5) from lOuchi et all d2008l) (dotted-dashed lines), 
IHu et al.1 d2010l) (plain lines) and iKashikawa etalj ( 1201 ll) (dot- 
ted lines). The transitions to thin lines indicate the range of the 
luminosities probed by the observations. A fixed faint end slope 
of - 1 .5 is assumed. The parameter space probed by the HAWK-I 
observations is delimited by the black line, with the black dashed 
line corresponding to the HAWK-I observations and th e null 
detection of the bright candidates of iHibon et al.1 (1201 Oh men- 
tioned in the text. The gree n dotted l ine co rresponds to a 60% 
evolution in density of the IHu et a ll (120101) LF, and the short- 
dashed-long-da shed line cor r espon ds to a 60% evolution in lu- 
minosity of the lOuchi et"aH d2010l) z = 6.5 LF. The red filled 
diamond corresponds to the live et al.l (120061) LAE detection at 
a redshift of 6 . 96. Th e black open diamonds correspond to the 
IVanzella etail d2011l) LAE detections at redshifts of 7.008 and 
7. 109. The magenta downward-pointing filled (resp. open) trian- 
gles ar e the photometric candi dates at z = 7.7 from iTilvi et all 
Izolol) (resp. iKmget "all (l20!"it) ). 



(l2010l) ; lKashikawa et al.1 d201 lh ). The large differences between 
the published LFs at z = 6.5 prevent us from inferring a robust 
estimate of the Lya IGM transmission at z = 7.7, and therefore 
of the neutral-h ydrogen fraction ^hi at this redshift. However, in 
all cases but the iHu et al.1 (l2010h LF, a significant quenching of 
the Lya transmission by the IGM, probably due to reionization, 
is required. 
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